Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights

A two-layer neural network can be used to approximate any nonlinear function. T h e behavior of the hidden nodes tha t allows the network to do this is described. Networks w i th one input are analyzed first, and the analysis is then extended to networks w i t h mult iple inputs. T h e result of th is analysis is used to formulate a method for ini t ial izat ion o f the weights o f neural networks to reduce t ra in ing t ime. Training examples are given and the learning curve for these examples are shown to illustrate the decrease in necessary training t ime. Introduction Two-layer feed forward neural networks have been proven capable of approximating any arbitrary functions [l], given that they have sufficient numbers of nodes in their hidden layers. We offer a description of how this works, along with a method of speeding up the training process by choosing the networks’ initial weights. The relationship between the inputs and the output of a two-layer neural network may be described by Equation (1) H l y = wi . sigmoid(LqX + W b i ) (1) i=O where y is the network’s output, X is the input vector, H is the number of hidden nodes, Wi is the weight vector of the i th node of the hidden layer, Wbi is the bias weight of the ith hidden node, w i is the weight of the output layer which connects the i th hidden unit to the output. The behavior of hidden nodes in two-layer networks with one input To illustrate the behavior of the hidden nodes, a two-layer network with one input is trained to approximate a function of one variable d(z) . That is, the network is trained to produce d(z) given z as input using the back-propagation algorithm [2]. The output of the network is given as